Computer Vision Technology Applied to MR-Based Pre-visualization in Filmmaking

In this talk, we introduce the outline of“ The MR-PreViz Project”performed in Japan. In the pre-production process of filmmaking, PreViz, pre-visualizing the desired scene by CGI, is used as a new technique. In its advanced approach, we propose MR-PreViz to utilize mixed reality technology as in current PreViz. MR-PreViz makes it possible to merge the real background and the computer-generated humans and creatures in an open set or at an outdoor location. Computer vision technologies are required for many aspects of MR-PreViz. For capturing an actor’s action, we applied 3D Video, which is a technology that allows one to reconstruct an image seen from any viewpoint in real time from video images taken by multiple cameras. As the other application of CV, we developed a vision based camera tracking method. The method collects environmental information required for tracking efficiently using a structure-from-motion technique before the shooting. Additionally, we developed a relighting technique for lighting design of MR-PreViz movie.